In recent years, technology development related to mixed reality (MR) is flourishing for the purpose of seamless merging between a physical world and a virtual world. MR has received a great deal of attention as a technique of reinforcing VR for the sake of coexistence of a physical world and a virtual reality (VR) world which can be experienced conventionally only in a situation separated from the physical space.
A typical device to implement mixed reality is a HMD (Head Mounted Display). This device implements mixed reality by compositing the physical space and virtual space and displaying them on the HMD.
In a system for supporting conferences or various cooperative operations between participants in remote sites by using VR, a world separated from the physical world is built in a computer and shared, as described above. On the other hand, a remote control system based on MR can support operations in a work space formed by superimposing a virtual world on the physical world.
An example of a remote control technique using a HMD is Japanese Patent Laid-Open No. 2002-132487 (US-2002-0049510) . In the technique disclosed in this prior art, an operator wears a HMD with a camera. The operator and an instructor in a remote site share an image from the camera. An instruction of the instructor is superimposed on the camera image to instruct an operation. In this technique, the operator's camera is a stereo camera so that he/she and the instructor can observe the same 3D image. A hand of the instructor pointing to the operation target in the image can be extracted by chromakey composition and composited with the image from the camera. In this system, no coordinates are set in the operator space. For this reason, any virtual object except the operation target can be neither laid out at an arbitrary position of the operator space nor pointed to cause interaction. Additionally, if the viewpoint position of the operator moves, the image of the hand also moves even when the instructor does not move his/her hand.
Another example of a technique of giving an instruction from a remote site to an operator who wears an optical see-through HMD with a camera, like the above-described known technique, is Block Party (Edited by W. Barfield and T. Caudell, “Fundamentals of Wearable Computers & Augmented Reality”, pp. 557-563 Lawrence Erlbaum Associates, Publishers, (2001)). Block party has a function of superimposing a CG image on a work space that the operator sees through the optical see-through HMD. The instructor grasps the progress of operation on the basis of images from the camera on the operator's head and supports the operation while manipulating the CG of the 3D model of the operation target. In this system, the camera image of the operator is not stereoscopic. No CG image is superimposed, either. The instructor sees the camera image displayed on the monitor screen of a desktop and manipulates the CG by using a 3D graphics editor. For this reason, he/she can hardly perceive the operator space as a seamless space and has no means for three-dimensionally pointing the camera image of the operator. In addition, the operator has no function of pointing or manipulating the CG.
In a system disclosed in U.S. Pat. No. 6,708,142, participants are sensed by stereo cameras, and the stereoscopic images are transmitted to other participants in remote sites so that the participants who wear HMDs can observe and manipulate a 3D image together with a virtual object shared between them. In this example, a participant is sensed by two stereo cameras, and the plurality of remaining participants can observe the 3D image. Since the position of the operator and the position of the hand to manipulate the virtual object are measured, manipulation of the virtual object is possible. However, the stereoscopic image observed by the participants includes the image of the participants without the background and the image of the VR space by the virtual object and has no information of the real space of the participants.